2 research outputs found
AltUB: Alternating Training Method to Update Base Distribution of Normalizing Flow for Anomaly Detection
Unsupervised anomaly detection is coming into the spotlight these days in
various practical domains due to the limited amount of anomaly data. One of the
major approaches for it is a normalizing flow which pursues the invertible
transformation of a complex distribution as images into an easy distribution as
N(0, I). In fact, algorithms based on normalizing flow like FastFlow and
CFLOW-AD establish state-of-the-art performance on unsupervised anomaly
detection tasks. Nevertheless, we investigate these algorithms convert normal
images into not N(0, I) as their destination, but an arbitrary normal
distribution. Moreover, their performances are often unstable, which is highly
critical for unsupervised tasks because data for validation are not provided.
To break through these observations, we propose a simple solution AltUB which
introduces alternating training to update the base distribution of normalizing
flow for anomaly detection. AltUB effectively improves the stability of
performance of normalizing flow. Furthermore, our method achieves the new
state-of-the-art performance of the anomaly segmentation task on the MVTec AD
dataset with 98.8% AUROC.Comment: 9 pages, 4 figure
Modality-Agnostic Self-Supervised Learning with Meta-Learned Masked Auto-Encoder
Despite its practical importance across a wide range of modalities, recent
advances in self-supervised learning (SSL) have been primarily focused on a few
well-curated domains, e.g., vision and language, often relying on their
domain-specific knowledge. For example, Masked Auto-Encoder (MAE) has become
one of the popular architectures in these domains, but less has explored its
potential in other modalities. In this paper, we develop MAE as a unified,
modality-agnostic SSL framework. In turn, we argue meta-learning as a key to
interpreting MAE as a modality-agnostic learner, and propose enhancements to
MAE from the motivation to jointly improve its SSL across diverse modalities,
coined MetaMAE as a result. Our key idea is to view the mask reconstruction of
MAE as a meta-learning task: masked tokens are predicted by adapting the
Transformer meta-learner through the amortization of unmasked tokens. Based on
this novel interpretation, we propose to integrate two advanced meta-learning
techniques. First, we adapt the amortized latent of the Transformer encoder
using gradient-based meta-learning to enhance the reconstruction. Then, we
maximize the alignment between amortized and adapted latents through task
contrastive learning which guides the Transformer encoder to better encode the
task-specific knowledge. Our experiment demonstrates the superiority of MetaMAE
in the modality-agnostic SSL benchmark (called DABS), significantly
outperforming prior baselines. Code is available at
https://github.com/alinlab/MetaMAE.Comment: Accepted to NeurIPS 2023. The first two authors contributed equall